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ABSTRACT 


ami  I4amiity  Sr  W#e*  numbm) 

fA  computer  program  is  presented  which  simulates  a Probability  Ratio  Sequential 
Test  (PRST)  plan  from  MIL-STD  781C.  Upon  selection  of  an  applicable  test  plan 
and  a lower  test  WTBF  Gj9j))  , the  simulation  determines,  for  a chosen  range  of 
true  WTBFs , itfT  the  probability  of  reaching  maximum  (total)  test  time  before 
making  a decision  (to  either  accept  or  reject  equipment)  and  (3)^  the  proba- 
bility of  reaching  the  last  failure  in  the  test  plan.  Examples  are  available 
from  selected  test  plans. 
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SIMULATION  OF  SEQUENTIAL  TESTS 


1 . INTRODUCTION 

The  computer  program  presented  in  this  paper  is  offered  as 
an  aid  to  test  planners  and  those  who  are  concerned  with  the  application 
of  the  Reliability  Design  Qualification  and  Production  Acceptance  Tests 
(Exponential  Distribution),  more  specifically,  the  Probability  Ratio 
Sequential  Test  (PRST)  plans  from  MIL-STD-781C. 

The  PRST  plans,  which  come  under  the  category  of  statistical 
test  plans,  should  be  employed  when  a sequential  test  plan  with  minimal 
decision  risks  (10  to  20  percent)  is  desired.  In  the  main,  the  PRST 
plans  would  be  preferred  to  the  fixed  length  test  plans  when  the  objec- 
tive of  the  test  is  to  accept  material  with  a high  mean-time-between- 
fai lures  (MTBF)  or  reject  material  with  a very  low  MTBF  as  quickly  as 

possible. * 


In  view  of  these  considerations,  the  utilization  of  the  metho- 
dology proposed  should  permit  the  test  planner,  equipped  with  a reason- 
able amount  of  practical  experience  with  the  PRST  plans,  to  make  certain 
probabalistic  statements  regarding  termination  points  in  the  plans, 
namely  (1)  the  likelihood  of  reaching  the  last  failure  and  (2)  the  likeli- 
hood of  reaching  the  maximum  test  time. 

Historically,  the  PRST  plans  have  no  provision  for  establishing 
a definitive  estimate  of  the  true  MTBF  of  an  item  prior  to  testing. 
Therefore,  the  expected  time  required  for  test  completion  may  vary 
significantly.  Consequently,  program  costs  and  schedules  have  to  be 
planned  to  compensate  for  this  range  of  uncertainty.  However,  with 
the  help  of  the  methodology  delineated  herein,  one  may  be  able  to  choose 
an  appropriate  test  plan  from  MIL-STD-781C,  select  a lower  test  MTBF  (0i), 
specify  a realistic  range  of  true  WTBFs  for  consideration,  implement  the 
simulation,  and  finally  obtain  measures  of  the  two  likelihood  estimates 
aforementioned,  all  of  this  in  order  to  reduce  the  range  of  uncertainty 
and  therefore,  minimize  program  cost  overruns. 

2.  APPLICATION  OF  MIL-STD-781C  AND  THEORETICAL  CONSIDERATIONS 

A typical  PRST  plan  from  MIL-STD-781C,  regardless  of  the  total 
number  of  failures  and  the  total  test  time  in  the  plan,  will  basically 
assume  an  appearance  as  depicted  in  Figure  1.  Given  that  the  decision 
risks,  discrimination  ratio,  total  number  of  failures,  and  accept-reject 
criteria  are  all  in  harmony,  of  concern  is  not  the  general  shape  or 
length  of  the  PRST  arrow  (shaded  area)  but  the  two  termination  vectors 
located  in  the  arrowhead.  For  simplicity,  we  may  designate  the  last 
failure  as  F^  and  the  maximum  total  test  time  as  T^.  The  problem,  then, 

^Proposed  MIL-STD-781C  Reliability  Design  Qualification  and  Production 
Acceptance  Tests:  Exponential  Distribution. 
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Figure  1.  A Generalized  PRST  Plan. 
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Figure  5.  Curve  Smoothing  Scheme  (3- Point) 


becomes  the  following:  When  testing  an  item  of  equipment  with  the  PRST 
plans,  how  often  is  the  last  failure  reached?  In  other  words,  what  is  the 
likelihood  of  reaching  F^? 

In  a similar  manner,  when  testing  an  item  of  equipment  with  the 
PRST  plans,  how  often  is  the  maximum  test  time  reached?  That  is,  what 
is  the  likelihood  of  reaching  T^?  With  these  thoughts  in  mind,  the  pur- 
pose of  the  simulation,  then,  is  to  ascertain  how  often  F,  and  TM  are 
reached.  (Figures  2,  3). 

Once  arrays  of  these  likelihood  estimates  are  determined  for 
a representative  range  of  true  MPBFs,  it  may  be  feasible,  strictly  for 
the  benefit  of  the  CALCOMP  plots,  to  effect  minor  changes  to  these 
arrays  by  employing  a simple  weighting  scheme  (Figure  4).  By  allowing 
a particular  data  point  to  have  a weight  of  1/n,  successive  neighboring 
data  points  in  either  direction  can  be  assigned  weights  of  l/2n,  l/4n, 
l/8n  and  so  on.  This  scheme  essentially  permits  valuable  information  to 
be  used  from  data  points  in  proximity  (Figure  5,  6).  If  one  is  not  so 
inclined  toward  implementing  such  curve -smoothing  techniques,  the  unsani- 
tized raw  data  points  are  still  available. 

3.  PROCEDURE  FLOWCHART 

All  that  the  user  of  the  computer  program  (henceforth,  referred 
to  as  program  MS  781C)  need  be  concerned  with  is  contained  in  this  sec- 
tion. An  adherence  to  the  principles  set  forth  in  the  procedure  flow- 
chart (Figure  7)  and  the  preparation  of  the  data  input  cards  for  program 
MS  781C  (Table  2)  should  permit  the  user  to  investigate  a wide  variety 
of  options.  There  are  two  areas  regarding  input,  though,  that  may  require 
clarification  - (1)  the  test  plan  matrix  and  (2)  the  accept-reject  cri- 
teria. 

A summary  of  the  test  plan  matrix  for  MIL-STD-781C  is  shown 

below. 

TABLE  1.  MILITARY  STANDARD  781  C 


The  Minimum  Number 
of  Failures  From 


Test  Plan 
Number 

Maximum  Number 
of  Failures 

Maximum  Test  Time 
(Theta  One  Multiple) 

Which  To  Reach 
Maximum  Test  Time 

1 

41 

49.50 

36 

2 

19 

21.90 

15 

3 

16 

20.60 

12 

4 

8 

9.74 

5 

5 

7 

10.35 

4 

6 

3 

4.50 

2 

7 

6 

6.80 

3 

8 

3 

4.50 

2 
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Figure  7 Procedure  Flow  Chart. 
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The  test  plan  matrix  (Table  1)  is  one  of  the  required  inputs 
to  program  MS  781C.  It  was  defined  in  this  way  so  that  the  user  could 
specify  the  number  of  entries  in  the  matrix  (up  to  25  entries  as  estab- 
lished in  the  program  logic).  Moreover,  the  test  plan  matrix  was  rele- 
gated to  input  status  in  order  to  provide  flexibility  for  any  user  who 
chooses  to  submit  a modified  version  of  the  test  plan  matrix,  a version 
based  on  innovative  truncation  points  perhaps. 

Since  the  accept -reject  criteria  must  be  in  accordance  with 
the  test  plan  matrix,  it  has  also  been  placed  in  the  domain  of  user- 
supplied  input  information.  Care  must  be  exercised,  though,  that  the 
values  for  the  reject  line  be  input  prior  to  the  values  for  the  accept 
line. 

4.  DESCRIPTION  OF  THE  ALGORITHM 

The  essence  of  program  MS  781C  resides  in  a section  of  code 
(the  algorithm)  in  the  heart  of  the  program.  This  set  of  instructions 
simulates  the  testing  of  an  item  of  equipment  using  a typical  PRST  plan. 
The  code,  that  is,  set  of  instructions,  makes  use  of  two  program  counters. 
One  counter  corresponds  to  the  last  failure,  F , and  the  other  counter 

corresponds  to  the  maximum  test  time,  T . During  each  iteration  of  the 

simulation,  an  item  of  equipment  can  be  either  rejected,  accepted, 
or  put  to  further  test  (as  exemplified  by  the  continue  test  strip, 

Figure  1).  If  the  item  is  rejected  or  accepted  prior  to  reaching  trunca- 
tion, then  no  counters  are  incremented,  and  a new  iteration  is  begun. 

If  the  item  falls  within  the  continue  strip  without  a rejection  or  an 
acceptance,  then  another  failure  time  is  called  for,  and  the  above 
process  is  repeated. 

In  order  to  test  against  the  maximum  test  time  value,  a particu- 
lar failure  time  is  compared  with  the  T^  value.  If  that  particular 

failure  time  is  greater  than  or  equal  to  the  TM  value,  then  the  TM  counter 

is  incremented  by  one  and  a new  iteration  is  begun.  If  that  particular 
failure  time  is  less  than  the  Tu  value,  then  another  failure  time  is 

called  for  an  queried  in  the  same  manner. 

Similarly,  in  order  to  test  against  the  last  failure,  a 
particular  failure  number  is  compared  with  the  maximum  failure  number, 

Fl,  to  determine  equality.  If  equality  exists,  then  the  F^  counter  is 

incremented  by  one  and  a new  iteration  is  begun.  If  equality  does  not 
exist;  that  is,  if  the  particular  failure  number  is  less  than  the  F^ 

value,  then  another  failure  is  called  for  and  queried  in  the  same 
manner. 

Now,  if  we  allow  each  iteration  to  be  an  independent  event, 
and  if  we  conduct  a large  number  of  these  trials,  then  the  F^  and  TM 
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counters  can  be  shown  to  be  likelihood  estimates  for  reaching  the  last 
failure  and  the  maximum  total  test  time,  respectively.  Furthermore, 
these  likelihood  estimates  can  be  viewed  in  probabalistic  terms. 

5.  RECOMMENDATIONS  FROM  COMPUTATIONAL  EXPERIENCE 

Once  a PRST  plan  has  been  chosen  and  a value  for  the  lower  test 
MTBF  (THETA  ONE)  has  been  designated,  the  only  values  that  may  require 
prudent  selection  are  the  three  input  parameters  BASE,  VALUE,  and 
DELTAX  (see  Table  2,  Appendix  1).  Clearly,  one  would  be  interested  in 
examining  a range  of  true  MTBFs  encompassing  both  the  lower  test  MTBF 
and  the  upper  test  MTBF  (THETA  ZERO).  Something  like  the  following 
might  be  of  interest: 


• » ■ • • 

BASE  6i  6 VALUE 

1 o 

A problem  may  surface  toward  the  lower  end  of  the  range  of  true 
MTBFs,  though,  if  a value  for  the  increment  (DELTAX)  is  chosen  too  large. 
This  problem  could  arise  in  subroutine  SUB  9 where  a neighborhood  of  nine 
(9)  points  is  operated  on: 

-5  0 5 10  15  20  25  30  35 

| — • a a a a • a • a 1 MTBF 

P1  P2  P3  P4  P5  P6  P7  P8  P9  (*n  ^ours) 


For  example,  with  a BASE  MTBF  of  15  hours  and  a DELTAX  increment 
of  5 hours,  the  "lower-end"  MTBF  could  attain  a negative  value,  thereby 
creating  a failure  mode.  However,  the  program  logic  is  set  up  to 
handle  this  problem  and  will  respond  with  a diagnostic  to  the  user. 

6.  ILLUSTRATIVE  EXAMPLES 

In  order  to  exemplify  the  range  of  possibilities  that  exists 


among 

; the 

PRST  plans, 

consider  test  plans 

II  and  VI  (Table 

3). 

A 

TABLE  3. 

COMPARISON  OF  TEST 

PLANS  II  AND  VI 

Decision 

Discrimination 

Total  Number 

Total  Test 

Risks 

Ratio 

of  Failures 

Time 

Test 

Plan 

II 

20% 

1.5:1 

19 

21.90*0! 

Test 

Plan 

VI 

20% 

3.0:1 

3 

4.50*0! 

18 


Both  test  plans  are  based  on  the  same  risks.  However,  they 
differ  in  at  least  two  respects:  Test  Plan  VI  has  twice  the  discrimination 
ratio  as  Test  Plan  II,  while  Plan  II  is  based  on  approximately  six  times 
as  many  failures  as  Plan  VI. 

Table  IV  below  demonstrates  simulation  results  for  both  plans 
at  two  specific  data  points  - THETA  ONE  and  THETA  ZERO.  In  each  cell, 
the  top  number  gives  the  probability  of  reaching  the  maximum  test  time, 

P (Tm) , while  the  bottom  number  gives  the  probability  of  reaching  the  last 

failure,  P - 

TABLE  4.  SIMULATION  RESULTS  FOR  TEST  PLANS  II  AND  VI 

@ THETA  ONE  @ THETA  ZERO 

P(Tm)  * .12  P(Tm)  - .21 

P(Fl)  * .12  P(Fl)  - .08 


P(Tm)  » .13  P(Tm)  = .39 

P(Fl)  » .86  P(Fl)  - .38 


The  purpose  of  this  study  has  been  to  develop  an  algorithm 
that,  with  the  aid  of  Monte  Carlo  simulation  techniques,  would  allow 
one  to  formulate  certain  probabalistic  statements  regarding  termination 
points  in  the  PRST  plans.  The  information  derived  from  program  MS  781C 
coupled  with  empirical  data  from  practical  field  experience  with  the 
plans  should  permit  the  test  planner  to  obtain  a grasp  on  the  likelihood 
of  arriving  at  the  concluding  points  in  the  plans.  With  this  kind  of 
information,  the  test  planner  may  be  able  to  minimize  program  costs 
due  to  personnel  and  materials  and  perhaps  get  a better  handle  on 
structuring  test  schedules. 


II 


TEST 

PLAN 


VI 
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APPENDIX  I 

TABLE  2.  DATA  INPUT  CARDS 


Data  Card  1 


Field 

1 

MAXNRF  (110) 

The  maximum  number  of  failures  for 
the  test  plan. 

Field 

2 

NRITER  (110) 

The  number  of  iterations  for  the 
simulation  e.g.,  2000  or  10,000. 

Field 

3 

NRTP  (110) 

The  number  of  the  test  plan  being 
implemented. 

Field 

4 

NRPLNS  (110) 

The  number  of  plans  in  the  test 
plan  matrix  (not  to  exceed  25) . 

Field 

5 

BASE  (F10.2) 

The  lowest  MTBF  in  the  range  of 
MTBFs  considered  e.g.,  if  an  MTBF 
range  of  50-250  is  desired,  then 
BASE  would  be  50. 

Field 

6 

VALUE  (F10.2) 

The  highest  MTBF  in  the  range  of 
MTBFs  considered. 

Field 

7 

DELTAX  (F10.2) 

The  increment  value  between  succes 
sive  MTBFs. 

Field 

8 

THETA1  (F10.2) 

The  lower  test  MTBF. 

Data  Card 

2_ 

This  data  card  consists  of  the  array  containing  the  values  for 
the  reject  line.  Data  must  be  arranged  sequentially,  eight  (8)  values 
per  card,  not  to  exceed  200  values,  i.e.  25  cards. 


Data  Card  3 

This  data  card  consists  of  the  array  containing  the  values  for 
the  accept  line.  Data  must  be  arranged  sequentially,  eight  (8)  values 
per  card,  not  to  exceed  200  values,  i.e.  25  cards. 

Data  Card  4 


• 

Field  1 

MATRIX  (K,  1) 
(F10.2) 

The  maximum  number  of  failures  for 
the  test  plan. 

Field  2 

MATRIX  (K,  2) 
(F10.2) 

The  maximum  test  time  (THETA  ONE 
Multiple)  for  the  test  plan. 

Field  3 

MATRIX  (K,  3) 
(F10.2) 

The  minimum  number  of  failures  from 
which  to  reach  maximum  test  time. 

Note:  For  data  card  4,  the  data  cards  must  be  arranged  sequentially,  i.e. 
the  first  card  must  begin  with  test  plan  1,  the  second  card  must 
begin  with  test  plan  2,  and  so  on.  Data  must  be  arranged  three 
(3)  values  per  card,  not  to  exceed  25  cards. 
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TABLE  5.  DICTIONARY  OF  PROGRAM  VARIABLES 


Variable 

Type 

Definition 

1 

NOF  CD 

Program/Output 

The  number  of  failures  in  the 
test  plan. 

2 

TMTBF  (I) 

Program/Output 

The  array  of  true  MTBFs  con- 
sidered . 

3 

D (I) 

Program 

An  array  containing  the  lowest 
and  highest  true  MTBF.  Finds 
utility  in  the  Fix  Scale 
CFIXSCA)  plotting  subroutine. 

4 

TITLE  (I) 

Program 

The  arrays  containing  the  sym- 
bol strings  for  the  plotting 
subroutines. 

5 

AKOUNT  (I) 
BKOUNT  CD 
CKOUNT  CD 
DKOUNT  CD 

Program 

ii 

ii 

ii 

Arrays  used  for  holding  values 
from  the  counters. 

6 

MAXNRF 

Input 

The  maximum  number  of  failures 
for  the  test  plan. 

7 

NRITER 

Input 

The  number  of  iterations  for 
the  simulation. 

8 

NRTP 

Input 

The  number  of  the  test  plan 
being  implemented. 

9 

NRPLNS 

Input 

The  number  of  plans  in  the 
test  plan  matrix. 

10 

BASE 

Input 

The  lowest  MTBF  in  the  range 
of  MTBFs  considered. 

11 

VALUE 

Input 

The  highest  MTBF  in  the  range 
of  MTBFs  considered. 

12 

DELTAX 

Input 

The  increment  value  between 
successive  MTBFs. 

13 

THETA 1 

Input 

The  lower  test  MTBF. 

APPENDIX  II 


TABLE  5.  DICTIONARY  OF  PROGRAM  VARIABLES  (CONTINUED) 


Variable 

Type 

Definition 

14 

INTVLS 

Program 

The  number  of  intervals  for 
the  accept-reject  criteria. 

15 

NRMTBF 

Program 

The  number  of  MTBFs  in  the 
range  of  MTBFs  considered. 

16 

UPPER 

Program 

The  value  used  for  maximum 
test  time. 

17 

LIMIT 

Program 

The  minimum  number  of  failures 
from  which  to  reach  maximum 
test  time. 

18 

KOUNT 

Program 

A local  counter  used  to  store 
the  number  of  times  maximum 
test  time  is  reached. 

19 

KNT 

Program 

A local  counter  used  to  store 
the  number  of  times  the  last 
failure  is  reached. 

20 

NCASE 

Program 

A local  counter  for  tracking 
the  number  of  cases  in  the 
simulation. 

21 

ACCUM 

Program 

An  accumulator  used  for  storing 
an  (exponential)  time  to  failure. 

22 

NFAIL1 

Program 

The  number  of  the  failure  being 
scrutinized  in  the  simulation. 

23 

RLINE  (I) 

Input/Output 

The  array  containing  the  values 
for  the  reject  line. 

24 

ALINE  (I) 

Input/Output 

The  array  containing  the  values 
for  the  accept  line. 

25 

MATRIX  (I) 

Input/Output 

An  array  containing  the  test 
plan  matrix. 
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Figure  8.  Curve  Smoothing  Scheme  ( 5-  Point ) 
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Figure  9.  Curve  Smoothing  Scheme  (7- Point) 
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APPENDIX  V 

SUMMARY  OF  RESULTS  - TEST  PLANS  I - VIII 


The  following  table  and  figures  give  a brief  summary  of  results 
for  test  plans  I through  VIII.  In  each  case,  THETA  ONE  has  been  chosen 
as  100  hours  MTBF.  Figures  la  - Villa  depict  the  probability  of 
reaching  the  last  failure,  P(FL),  while  Figures  lb  - VUIb  depict  the 

probability  of  reaching  the  maximum  test  time  , P(TM). 
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